DOCUMENT RESUME 



ED 470 088 



IR 021 526 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Grabowski, Barbara L.; Koszalka, Tiffany A. 

Integrating Assessment and Research Strategies on a Large 
Development and Research Project: Kids as Airborne Mission 
Scientists (KaAMS). 

2001 - 11-00 

7p.; In: Annual Proceedings of Selected Research and 
Development [and] Practice Papers Presented at the National 
Convention of the Association for Educational Communications 
and Technology (24th, Atlanta, GA, November 8-12, 2001) . 
Volumes 1-2; see IR 021 504. 

Reports - Research (143). — Speeches/Meeting Papers (150) 

EDRS Price MFOl/PCOl Plus Postage. 

^Educational Assessment; Educational Resources; Elementary 
Secondary Education; Instructional Design; Instructional 
Development; Instructional Materials; Problem Based Learning; 
^Research and Development; ^Science Activities; ^Science 
Education 



ABSTRACT 

Combining assessment and research components on a large 
development and research project is a complex task. There are many 
descriptions of how either assessment or research should be conducted, but 
detailed examples illustrating integration of' such strategies in complex 
projects are scarce. This paper provides definitions of assessment, 
evaluation, and research and illumination on how to strategically integrate 
and manage assessment and research activities by providing examples of tools 
used to develop and manage a comprehensive assessment and research plan for 
Kids as Airborne Mission Scientists (KaAMS) , a large instructional 
development and research project. The purpose of the KaAMS project was to 
develop a series of teacher resources, framed in interdisciplinary problem- 
based. learning approaches integrating authentic and ill-structured problem 
situations, inspiring teachers to inspire students to pursue the sciences. 

The goal of the project was to develop informed instructional materials and 
assess their impact, on the target audiences, middle school teachers and 
students. Thus the project proposal included a strong commitment to formative 
and summative evaluation as well as a research component." (Author/AEF) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. . 



n® IR021526 



Integrating Assessment and Research 
Strategies on a Large Development 
and Research Project: Kids as 
Airborne Mission Scientists (kaAMS) 



By: Barbara L. Grabowski & Tiffany A. Koszalka 






PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

P 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

0 This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 






1 



V 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 






• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




2 



Integrating Assessment and Research Strategies On a Large Development and 
Research Project: Kids as Airborne Mission Scientists (KaAMS) 

Barbara L, Grabowski. 

Penn State University 

Tiffany A, Koszalka 
Syracuse University 



Abstract 

Combining assessment and research components on a large development and research project is a complex task. There are 
many descriptions of how either assessment or research should be conducted, but detailed examples illustrating integration of 
such strategies in complex projects are scarce. This paper provides definitions of assessment, evaluation, and research and 
illumination on how to strategically integrate and manage assessment and research activities by providing examples of tools 
used to develop and manage a comprehensive assessment and research plan for Kids as Airborne Mission Scientists (KaAMS). a 
large instructional development and research project. 



Introduction 

Kids as Airborne Mission Scientists (KaAMS) is a large development and research project funded by the National 
Aeronautics and Space Administration’s (NASA) Leading Educators to Applications, Research, and NASA-Related Educational 
Resources in Science (LEARNERS) project. The purpose of the KaAMS project was to develop a series of teacher resources, 
framed in interdisciplinary problem-based learning approaches integrating authentic and ill-structured problem situations, 
inspiring teachers to inspire students to pursue the sciences. The goal of the project was to develop informed instructional 
materials and assess their impact on the target audiences, middle school teachers and students. Thus, the project proposal 
included a strong commitment to formative and summative evaluation as well as a research component. 

Definitions 

Assessment is an umbrella term that refers to the systematic study of a problem or innovation to make effective decisions 
about what should happen (Rossett, 1987). In the case of the KaAMS project, the assessment plan defined the objectives, 
strategies, tools, and protocols used to collected formative and summative feedback. 

The formative evaluation component of the assessment plan answered the question bow are we doing?’ as instructional 
materials were being developed (Morrison, Ross & Kemp, 2001). The focus of formative evaluation was on measuring the 
effectiveness, efficiency, usability, and acceptability of the material produced so that sound instructional design decisions could 
be made as the materials were developed and finalized. Feedback collected during formative evaluation served the function of 
informing the development team how well each of the instructional materials were serving the instructional objectives and what 
should be done to enhance them. 

The summative evaluation procedures were designed to answer the question ‘how did we do?’ after instructional materials 
had been implemented with the target audience (Morrison, Ross & Kemp, 2001). Summative feedback focused specifically on 
the degree to which the major instructional outcomes were attained as a result of using KaAMS materials. Summative evaluation 
addressed (1) reaction, (2) learning gains, (3) performance changes, (4) education system changes, and (5) impact on the greater 
society. This feedback was useful in developing implementation plans and supporting materials to train new teachers in the use of 
KaAMS materials. 

Research, on the other hand, is a process of identifying something unknown and then collecting data to make it known (Gall, 
Borg & Gall, 1996). In the case of KaAMS, the research components focused on analyzing the effect that using KaAMS had on 
the stakeholders in the middle school learning environment. For example, it was unknown how teaching strategies would be 
affected by introducing KaAMS into the classroom or whether the use of these materials would affect student interest in pursuing 
science. Answering such questions can help researchers develop a better understanding of the types of materials and activities 
that affect instructional and learning processes. 

Thus, the purpose of assessment is to gather feedback that will inform the instructional design process. The dangers of having 
incomplete and poorly thought out assessment strategies are that either required feedback is not collected or time is wasted 
collecting feedback that is not important to design -enhancement decisions. Thus, the focus of a good assessment strategy must 
include plans and instruments for collecting feedback that is essential to drawing conclusions about what is needed to develop 
great instruction. The purpose of research is to gather data on the impact that instructional materials have on stakeholders and the 
instructional and learning environment. Poor planning of the research data collection process can result in obsolete data, lost 
opportunities to collect needed data, or data collection conditions that interfere with gathering reliable data. Thus, strategic 
collection of research data must also be tied to the development process, based on the key research questions, provide flexibility 
for adjusting to new learnings during product development phases, and seek data that can test theoretical assumptions. 
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Although assessment and research literature provides guidelines to develop, manage, and conduct assessment or research, 
comprehensive examples illustrating how to integrate assessment and research strategies, methods, and tools could not be found 
in the literature. Published literature on large-scale development and research projects usually contained only brief descriptions of 
certain components of assessment and research often neglecting to provide detailed descriptions of the relationship among 
development cycles, feedback and data collection procedures, and assessment and research tools. Therefore, decision-making 
about design and implementation issues can be problematic if the right amount or right kinds of feedback and data are not 
collected. Examples that illustrate the integration of assessment and research strategies and tools are scarce. Sharing such 
examples is therefore beneficial to others developing comprehensive and targeted assessment and research plans. The remainder 
of this paper will describe and provide examples of the tools and procedures developed to manage and conduct assessment and 
research on the KaAMS project. 

Strategic Planning of Assessment and Research Cycles 

The challenge in creating a strategic approach to assessment and research began with identifying the formative and research 
needs of the project. The literature on instructional development formative and summative evaluation provided guidelines for 
determining the types of feedback needed to develop sound instructional materials (Dick and Cary, 1985). The scope of the 
KaAMS project helped to detamine who, e.g., teachers, administrators, students, curriculum specialists, would best be able to 
provide such feedback. Thus, during the development of the KaAMS lesson plans feedback was sough from middle school 
administrators, teachers, and students on the layout of the material, background resources, terminology, appropriateness of 
activities, time dedicated to preparation, impressions after using the materials, and thoughts on what worked and what did not 
during the use of the materials. This feedback was gathered during and immediately after the initial classroom testing of KaAMS. 

The research questions were drawn in-part from the purpose of the grant, a literature review on problem-based learning, 
technology integration in the schools, and career development in adolescents and in-part based on the researchers interests. The 
research questions probed for understanding on the effects of KaAMS on: (1) teachers use of KaAMS and NASA resources in 
their classrooms, (2) changes in teaching practices during and after using KaAMS materials, (3) changes in students’ success 
rates in science, (4) changes in students’ interests in pursuing science, and (5) diffusion of KaAMS and NASA materials to the 
surrounding educational environment. 

Three cycles of development and data collection 

Being a large development project, lasting three years and consisting of collaborations among two major universities, several 
NASA scientists, and several schools, the KaAMS products were developed in three cycles: alpha, beta, and final versions. The 
frameworks for the PBL units and major lesson plan components were developed during the alpha cycle. The alpha version was 
developed based on collaboration among the KaAMS project team that included a group of advising middle school teachers who 
reviewed and classroom tested the KaAMS materials. Assessment of the alpha materials focused on the appropriateness of the 
level of content and activity for the students, background information and instruction for teachers, and general organization of 
lesson plans and associated lesson components. 

The beta cycle focuses on using developing a beta version of KaAMS that incorporates learnings from alpha testing, 
including modifications needed to activities and development of support materials to meet the needs of teachers. Beta testing is 
conducted with a new set of teachers from a broader group of teachers and students spread across three states. Beta testing 
focuses on the usefulness of the completed product and how teachers integrate the lesson plans or lesson components into their 
curriculum. This feedback helps to finalize the KaAMS products and teacher support resources and develop national distribution 
plans. Simultaneously, research data were, or will be, collected to establish measurement baselines and gather further data on the 
effects of KaAMS on teachers, students, and the surrounding community. During Alpha testing the research instruments, 
protocols, and logistics were be tested. 

The assessment and research matrix 

The first step in developing the overall assessment plan was to document and examine the assessment and research needs. 
The assessment and research questions were placed in a matrix to develop a strategic view of the feedback and data requirements 
for the project. The formative and summative feedback needs were cross-referenced to the data collection instruments and 
questions by subject, e.g., teacher, student. See Table 1. 
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Table 1. Example from KaAMS Research and Assessment Matrix: Lesson Plan Development - Formative Evaluation 



Lesson Plan Development - 


Formative Evaluation 






Stakeholders/ Instruments / 
questions 


impressions of ^ , a c 

, ^ value / needs of 

, background 

terminology, etc. 

, resources 

of lesson plans 


connections to the 

curriculum, 

terminology 


value / usefulness / 
appropriateness of 
activities and 
resources 


TEACHERS 






J 


TAP A initial interview 








Teacher background 
Curriculum connections 


Q4-5 


Ql-3 




Lesson plan impressions 


Q6-7 




Q8 


Background content 


QlO-13 






Classroom use 
Discuss KaAMS w/others 


Q15 






LP Follow-up survey 








lesson used 


Q4 




Q5 


resources used 






Q8-9 


classroom description 
during lesson 


Q18 


Q17,Q23-24 


Q 19-22 


student success 









STUDENTS 



Student focus groups XXX X 

Student classroom 

documents X X 

Student journals X X 

* B=baseline, A=immediately after use, L= long-term follow-up, 1 to 6 months after use P-During prep, Q=question 

The research questions were also cross-referenced to the data collection instruments and questions by subject, e.g., teacher, 
student. See Table 2. 

After cross-referencing all of the feedback and data needs by instrument and stakeholder the matrix was examined for gaps 
and overlap. Instruments and protocols were adjusted to collected feedback and data that were not accounted for in the matrix. 
For example, it was found that student opinion and artifacts were gathered to assess student success rates, but teacher feedback 
was not gathered on student success. Questions were added to interview protocols to gather such feedback. 

Analysis of the overlaps indicated that there were several instruments and questions gathering feedback on the materials. 
Since this goal of the project was to develop new instructional materials it was agreed that the level of overlap, especially since 
feedback was from multiple perspectives of the teachers and students, was appropriate. To ensure that the data collection was not 
too obtrusive to the learning environment, the next step was to plan the timing for administration (f the data collection 
instruments and protocols. 

The assessment strategies protocols called for continual gathering of feedback from teachers and students through multiple 
strategies, e.g. surveys, interviews, and observations. The KaAMS product development plan laid out the calendar timing for the 
development and testing of each component of the instructional materials. Examining each instrument and protocol provided an 
estimate of how long each stakeholder would be required to provide solicited feedback and data. Thus, the timing, by calendar 
date, and timing in terms of stakeholder commitment in providing feedback through the various instruments were examined. 
Time frames were established for administering data collection instruments based on the project development timelines and 
collection patterns for the most appropriate data that would be least invasive to those involved. These estimated timelines were 
added to the KaAMS Research and Assessment matrix. See Table 2, freq columns. 
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Table 2. Example from KaAMS Research and Assessment Matrix: Research 



KaAMS Research 











How are 




How are student 






How are 




How are 


student 




levels of interest in 






teachers using 




teachers 


success rates 




pursuing science- 




Stakeholders/ 


KaAMS and 


♦ 


changing their 


in science 


♦ 


related career 


♦ 


Instruments / 


NASA 


fre 


teaching 


* changing over 


fre 


changing over 


fre 


questions 


resources? 


q 


practices? 


freq time? 


q 


time? 


q 



TEACHERS 



Electronic survey 


Q103,Q114- 

115,Q122- 


BA 


demographics 


136 


L 


teacher attitude 


Ql-31 


BA 

L 


percept, of env. 




BA 


for web 


Q32-52 


L 


methods of 


Q53-78, 


BA 


teaching 


Q112-113 


L 




Q79-102, 

Q104-110, 


BA 


Resource use 


Qlll 


L 


Importance of 
NASA, Aero, RS 




BA 


to curric. 


Q116-121 


L 



LP Follow-up 
survey 
lesson used 

resources used Q8-9 

classroom 
description during 
lesson 

student success 
Classroom 

Observations X D X D_ 

* B=baseline, A=immediately after use, L= long-term follow-up, 1 to 6 months after use P-During prep, Q=question 

One additional benefit of the strategic matrix was in providing an overall picture of the project development, assessment, and 
research components. Thorough review of the matrix provided insights into the complexity of the research agenda. This 
prompted the development of a strategy to use the alpha testing cycle to test the research protocols. This would allow the team to 
refine logistics for data collection and identify initial findings related to the effects of using the KaAMS materials with middle 
school populations. However, it is important to report such findings with caution, since effects may be related to additional 
attention from product development teams. 

Additional Planning and Managing Tools 

The strategic planning matrix provided the framework for a comprehensive Assessment Plan to be used by the project team 
to manage the assessment and research processes. The components of the Assessment Plan included text and graphic 
representations of the overall assessment strategy, teacher advisory panels for alpha and beta testing, roles and responsibilities of 
assessment team members, components of each evaluation and research procedure, development and data collection timelines, 
selection criteria for participants, forms for tracking procedural issues, copies of all instruments and protocols, and human 
subjects approval forms. The Assessment Plan was developed so that each component was tied to the research matrix supporting 
the data collection needs while keeping the amount of disruption to subjects at a minimum. 

The strategic planning matrix also provided the framework for creating a database for all data collected during the assessment 
and research phases. In this case, a spreadsheet was developed with multiple worksheets. Each worksheet was titled based on the 
instrument, stakeholder and type of data it collected. This naming structure ultimately helped in the data analysis processes. 

The final assessment and research report was also based on the strategic planning matrix. The matrix provided an overview of 
the data requirements in response to the evaluation and research questions. This overview was used as the planning structure to 
analyze the data collected and to report on findings based on the project questions. The final assessment and research report 



Q10,Q16, 

Q27 A Q 19-22 A 

Q13,Q15 A 
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included text and graphic representations for an executive summary of the assessment and research findings, a short summary of 
the project goals and assessment aid research objectives, summary of the formative and summative evaluation feedback and 
recommendations for each component of the KaAMS material, tables and figures describing results, summaries of the 
observations and data collected from each classroom, baseline and subsequent description of each stakeholder in the classroom 
trials, and appendices that described each of the lesson plans tested, detailed notes from observations, interviews, and focus 
groups, and selected printouts of data analysis on stakeholders. Thus, the information in the summary report was linked to the 
other assessment and research tools developed for the project, based on the framework established in the strategic planning 
matrix. 

Conclusions 

Combining assessment and research components on a large development and research project is a complex task. Creating 
strategic planning tools can help visualize the feedback and data needs and provide a framework for creating overall assessment 
and research plans, data collection schedules and instruments, and summary reports. 

During KaAMS, several levels of assessment and research were conducted and data were collected from several stakeholders. 
A planning matrix was created to avoid over- and under-collection of data and assure that the appropriate data were collected. 
The matrix provided a strategic tool to help cross-reference the collection of different data, at different times, from multiple 
stakeholders. It helped to identify weaknesses and strengt hs of the assessment strategies and guided further development of 
instruments and protocols. The key benefit of such as matrix was in its value to provide a framework for creating an overall 
Assessment Plan that helped coordinate assessment and research team members during data collection, data analysis, and 
reporting. This tool, thus, helped to plan and manage the complex task of gathering the right type of feedback and data, at the 
right time, from the right stakeholders during a large, multi-year development and research project. 
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